1. Technical Field
Present invention embodiments relate to object storage systems, and more specifically, to a compute engine of a distributed object storage system supporting execution of computational algorithms within a specified time frame.
2. Discussion of the Related Art
A distributed object storage architecture comprises two entities (or node groups) including proxy nodes and storage nodes. Proxy nodes are used for distributed load handling/request handling into a storage namespace, while storage nodes are responsible for writing to disks/storage sub-systems. This architecture serves as a storage unit/repository, where analysis of the data residing in the storage nodes (e.g., extracting meaningful information from raw data) may be accomplished by an additional client or compute node.
An enhancement to this architecture comprises a compute engine present within the storage and/or proxy nodes. An end-user frames a computational algorithm and deploys or passes the computational algorithm to the compute engine as a normal object (Hypertext Transfer Protocol (HTTP)) PUT operation. Since the storage/proxy nodes act as compute nodes and return results to the end-user, this enhanced architecture does not require any additional client or compute node to perform analysis of the data.
In a synchronous mode of operation, the compute engine executes the computational algorithm within a Hypertext Transfer Protocol (HTTP) request/response that initiated the execution. In other words, the HTTP request ends after the computational algorithm completes execution. However, in an asynchronous mode of operation, the HTTP request that initiated execution of the computational algorithm completes as soon as the system registered the request in a queue. The computational algorithm is executed in the background, and results may be accumulated in an output object that further includes information about a completion status. Subsequently, an initiator of the computational algorithm may access the output object and retrieve the results of the computation.
In both the synchronous and asynchronous modes of operation, once the computational algorithm is deployed by the end-user, the compute engine receives the computational algorithm and invokes a container (whose resources are limited by the entitlements of the user) on a best node (any of the proxy or storage nodes, where the best node is selected based on the maximum available resources at that particular point of time). For the synchronous mode of operation, the end-user simply waits (e.g., at a terminal of an end-user device) for the completion of the computational algorithm (HTTP request completion). There is no indication of a wait time for completion of the computational algorithm.
With respect to the asynchronous mode of operation, the end-user continuously checks the output object for completion of the computational algorithm. There is no indication of the length of time to complete the computational algorithm, and the end-user cannot specify a maximum time for the computational algorithm to complete processing. For example, a distributed object storage system may operate with a lesser workload during non-business hours, and the administrator or end-user desires to finish processing of complex algorithms within this time window. However, the end-user does not have control of resources to time bound the computational algorithm.
Another scenario pertains to Quality of Service (QoS). In this case, an end-user may have a high QoS, and all nodes may be used for faster processing of a single computational algorithm. However, this is inefficient since all the nodes or computation resources are used even though the computational algorithm may be completed within a desired time frame (with a lesser quantity of nodes).